Learning from Finite Training Sets

{ We analyse online (gradient descent) learning of a rule from a nite set of training examples at non-innnitesimal learning rates , calculating exactly the time-dependent generalization error for a simple model scenario. In the thermodynamic limit, we close the dynamical equation for the generating function of an innnite hierarchy of order parameters using`within-sample self-averaging'. The resulting dynamics is non-perturbative in , with a slow mode appearing only above a nite threshold min. Optimal settings of for given nal learning time are determined and the results are compared with ooine gradient descent. Neural networks have been the subject of much recent research because of their ability to learn rules from examples. One of the most common learning algorithms is online gradient descent: The weights of a network (`student') are updated each time a training example from the training set is presented, such that the error on this example is reduced. In ooine gradient descent, on the other hand, the total error on all examples in the training set is accumulated before a gradient descent weight update is made. For a given training set and starting weights, ooine learning is entirely deterministic. Online learning, on the other hand, is a stochastic process due to the random choice of training example (from the given training set) for each update. It becomes equivalent to ooine learning only in the limit where the learning rate ! 0 1]. In both cases, the main quantity of interest is the evolution of the generalization error: After a given number of weight updates, how well does the student approximate the input-output mapping (`teacher' rule) underlying the training examples? We do not consider in the following non-gradient descent learning algorithms, and also restrict ourselves to gradient descent on the most common measure of error on a training example, the squared output deviation (see eq. (1) below). For interesting recent results on more general, optimized online learning algorithms, see, e.g., 2, 3].